Skip to content

AddFiles: PathPattern and RouteFiles - #40749

Open
claudevdm wants to merge 3 commits into
apache:masterfrom
claudevdm:addfiles-dynamic-dest2
Open

claudevdm wants to merge 3 commits into
apache:masterfrom
claudevdm:addfiles-dynamic-dest2

Conversation

@claudevdm

Copy link
Copy Markdown
Collaborator

Adds the two self-contained pieces that dynamic destinations for IcebergIO.AddFiles are
built on: a path-pattern matcher, and the DoFn that uses it to name a destination table
from each file's path.

AddFiles registers every input file into one table named by table. Migrating a lake
usually means a tree of per-table folders, which today needs one AddFiles per table.
The goal is for one transform to route each file to the table its path names:

path_pattern:  gs://lake/export/{table}/**
table:         default.{table}

gs://lake/export/orders/part-0.parquet     ->  default.orders
gs://lake/export/customers/part-0.parquet  ->  default.customers

This PR adds only the routing primitives so they can be reviewed on their own. The
pre-pass, registration and schema-transform surface follow separately.

PathPattern

A whole-path glob with {name} placeholders, each capturing exactly one path segment.
Globbing is Beam's own (FileSystemUtils.wildcardToRegexp), so *, ** and ? behave as
they do in any other Beam file pattern; placeholders are translated to capturing groups
between the globbed fixed parts.

The whole path must match, so the pattern decides both which table a file belongs to and
whether it belongs at all. compile rejects patterns that cannot answer that
unambiguously:

rejected reason
{a} twice a name can only be captured once
{a b}, {a.b} a capture is rendered into a dotted table identifier, so no dots, slashes or blanks
gs://lake/**/{tbl}/** a ** above a placeholder leaves the captured folder undetermined
gs://lake/{tbl}/ ends with /, so it names a folder and matches no file

render(template, captures) fills the table template, and checkRenderable(template)
refuses a template naming a placeholder the pattern never captures, so table: default.{db} against .../{tbl}/** fails at construction rather than producing one error
row per file.

RouteFiles

DoFn<String, KV<String, String>> with two outputs:

  • ROUTED: KV<table, path> for a file the pattern matched
  • ERRORS: a {file, error, table} row for a path it did not, with table empty because
    there is no table to attribute it to

An unmatched path becomes an error row rather than being silently dropped: a mistyped
pattern must be visible. PathPattern.compile and checkRenderable both run in the
constructor, so a bad pattern or template fails at pipeline construction instead of once
per file on a worker.

  • RouteFiles.ERROR_SCHEMA is declared locally here. When the transform is wired up it
    becomes the same schema as AddFiles' error output, so the two can be flattened; that
    change belongs with the wiring.
  • Patterns are interpreted by Beam's Java glob translator. Note it normalises **/* to
    **, which Python's FileSystem.translate_pattern does not. Relevant only if you
    compare a path_pattern against a glob written for Python's MatchFiles upstream.

@claudevdm
claudevdm requested a review from ahmedabu98 October 23, 2026 17:14
@claudevdm
claudevdm marked this pull request as ready for review October 23, 2026 17:14
@claudevdm

Copy link
Copy Markdown
Collaborator Author

R: @ahmedabu98

@github-actions github-actions Bot added java io iceberg Iceberg IO connector size/l L: 250-999 lines changed labels Oct 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

+ " determined; a '**' can only come after the last placeholder.",
pathPattern);
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this is too strict? The following template should be valid:

gs://exports/**/{table}/*.parquet

i.e. if the placeholder is the lowest directory

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Relaxed a bit so there can be at most one **.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But now that would reject something like gs://bucket/{table}/**/dir/**/*

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These rules are to identify the table membership/name from the {placeholder} parameter. I think we can assume that incoming files represent the datafiles that need to be processed . If we are not able to parse the placeholder from a file then it should be an error. User can just do gs://bucket/{table}/** to achieve the same.

MatchAllFiles pattern can be used to filter initial set of files.

I can try find a way to validate pattern with 2+ double star if you feel strongly though.

Comment thread sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RouteFiles.java Outdated
@claudevdm
claudevdm force-pushed the addfiles-dynamic-dest2 branch from fb03152 to 8f23d53 Compare October 26, 2026 20:14
@claudevdm
claudevdm requested a review from ahmedabu98 October 26, 2026 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

iceberg Iceberg IO connector io java size/l L: 250-999 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants